Approximating discrete probability distributions with Bayesian networks

I generalise the arguments of [Chow & Liu 1968] to show that a Bayesian network satisfying some arbitrary constraint that best approximates a probability distribution is one for which mutual information weight is maximised. I give a practical procedure for finding an approximation network. The plan is first to discuss the approximation problem and its link with Bayesian network theory. After identifying the optimal approximation, the bulk of the paper will be devoted to a proposed approximation algorithm which sacrafices optimality in favour of practicality. We shall investigate the theoretical basis and performance of this approximation strategy. 1 THE APPROXIMATION PROBLEM It can be quite unfeasible to determine, store and manipulate a discrete probability distribution. Working with a distribution over binary variables, for instance, one must find and store at least probability values and it may still take too long to calculate the probability of a statement (or event) of interest. Consequently the question of whether one can approximate a distribution sufficiently accurately using a reasonable number of specifying values is of key importance. [Chow & Liu 1968] partially answered this question using an innovative and highly practical method. I shall briefly outline their answer in the language of Bayesian network theory. A Bayesian network is a directed acyclic graph (dag) over nodes , together with a set of probability values for each node literal (of the form where , the values that node can take) and each state (a conjunction of literals ) of the direct predecessors or parents of . Under an independence asNote that for each node the probability of one literal can be sumption, namely that any node is independent of its nondescendants conditional on its parents, a Bayesian network suffices to determine a probability distribution, since for each atomic state of the form we have that where is the literal and is the parent state of node consistent with (the factor is taken to be the unconditional probability if has no parents). Thus, depending on the connectivity in the graph, the Bayesian network representation of a probability function can dramatically reduce the number of probability values required to specify the function. I shall call the number of specifying probabilities the complexity of a Bayesian network. While a complete dag on binary-valued nodes requires maximum complexity in a Bayesian network, a tree-shaped dag only requires specifiers, a significant saving. Chow and Liu showed that a tree-based Bayesian network (with induced probability distribution ) that best approximates a target distribution (according to the cross-entropy measure of distance between distributions where the are the atomic states) is a maximum weight spanning tree where arcs between two nodes are weighted by their mutual information